The present invention relates to a camera shake correcting device for correcting picture movement due to shaking of a video camera or the like by image processing.
Some of recent compact and light video cameras are equipped with a camera shake correcting device. As a countermeasure to the camera shake, there exist a camera shake preventing device for suppressing the occurrence of the camera shake by adopting a gyro or a high-speed shutter and a camera shake correcting device for correcting the camera shake having occurred by image processing.
FIGS. 7A to 7D illustrate a principle of camera shake correction. FIG. 7A shows a picture in an arbitrary field of a dynamic image signal series, the field being assumed as a first field. Reference character P denotes a point set at an arbitrary position in this picture. A picture element data at the point P will be represented by A(P).
FIG. 7B shows a picture in the next field, that is, in a second field. A point Q at which a picture element data is represented by B(Q) should be located at the point P, but the point Q is displaced to a position shown by a movement vector Vp because of camera shake.
FIG. 7C shows a picture obtained by overlapping the first field and the second field without correcting the camera shake. In this case, the picture becomes an inferior video image wherein both the point P and Q which should accord with each other are separated because of camera shake.
To cope with this problem, referring to FIG. 7B, a search area W including the point P as an origin is set, and a position occupied by the picture element data B showing a maximum correlation to the picture element data A at the point P in the search area W is searched. That is, the point Q is searched in the search area W. The search point Q showing the maximum correlation to the point P is assumed as a picture element corresponding point with respect to the point P, and the movement vector Vp is obtained from a distance and a direction from the point P (the origin) to the point Q.
FIG. 7D shows a picture wherein a video image position of the second field with respect to the first field is corrected by the movement vector Vp obtained above. By this correction, a good video image is obtained wherein the point P in the first field and the point Q in the second field accord with each other.
A representative point method is known as an example of detecting methods for the movement vector necessary for the camera shake correction.
FIG. 8 illustrates a conventional representative point method as the movement vector detecting method.
Referring to FIG. 8, Aij denotes a picture element data constituting the first field, and P0, P1, P2 and P3 denote representative points Pk provided in the first field. Picture element data at the representative points Pk are denoted by Ak. In this case, four points (k=0, 1, 2, 3) are selected as the representative points Pk for the convenience of illustration. Bij denotes a picture element data constituting the second field, and W0, W1, W2 and W3 denote search areas where Pk corresponding points in the second field are to be searched with respect to the representative points Pk as origins in the first field. Each search area Wk is so set as to include the corresponding representative point Pk as a search origin and occupy a horizontal range L and a vertical range M.
In case of obtaining the movement vector with the accuracy of .+-.a in the horizontal direction (H) and .+-.b in the vertical direction (V), search points Qk(l, m) are set in the search area Wk at the intervals of 2a(H) and 2b(V) as shown. That is, the accuracy of the movement vector depends on a density of the search points Qk(l, m) in the search area Wk. In the case that an input signal is a digital image data and the coordinates l and m of the search points Qk(l, m) are decimal, that is, each search point is located between picture elements, picture element data Bk(l, m) at the search points Qk(l, m) may be obtained from the adjacent picture element data by a known interpolation method. Accordingly, a distinction between analog image data and digital image data will not be particularly mentioned in the following description.
The movement vector of the second field with respect to the first field in the above case is obtained in the following manner. Absolute values of differences between the picture element data Bk(l, m) at the search points Qk(l, m) in the search area Wk and the picture element data Ak at the representative points Pk are summed up for k to obtain residuals S(l, m) as follows: EQU S(l, m)=.vertline.B0(l, m)-A0.vertline.+.vertline.B1(l, m) -A1.vertline.+.vertline.B2(l, m)-A2.vertline.+.vertline.B3(l, m)-A3.vertline..
As the total number of the search points Qk(l, m) in the search area Wk corresponding to the representative points Pk is L/(2a).times.M/(2b), the total number of the residuals S(l, m) to be obtained is also L/(2a).times.M/(2b). For example, the residuals S(l, m) are as follows: EQU S(0, 0), S(2a, 0), S(-2a, 0), S(0, 2b), S(0, -2b), . . . , S(2a, 2b), S(-2a, 2b), . . .
The minimum one S(l0, m0) of the L/(2a).times.M/(2b) residuals S(l, m) is obtained, and the picture element corresponding point in the second field with respect to the representative point Pk in the first field is set to Qk(l0, m0). In this way, the movement vector Vp is obtained from a distance and a direction from the search origin Pk to the corresponding point Qk(l0, m0).
In displaying the second field in succession to the first field, a display position of the second field is modified so that the representative point Pk in the first field and picture element corresponding point Qk(l0, m0) in the second field may be displayed at the same position on the picture plane according to the movement vector Vp, thereby correcting the picture movement due to the camera shake.
FIG. 9 is a block diagram of a conventional camera shake correcting device based on the above principle of correction of the camera shake.
Referring to FIG. 9, it is assumed that the picture element data Ak at the representative points Pk (k=0, 1, 2, 3) are stored in a representative point memory 11 during an inputting period of picture signals of the first field. Of the picture signals of the second field to be input, the picture element data Bk at the representative points Qk are stored through a switch SW1 into a representative point memory 12, and they become the picture element data Bk at the representative points Qk in a picture signal processing for the next or third field. At the same time, the picture signals of the second field to be input are supplied to an input terminal 211 of a residual calculating circuit 21 and a field memory 41.
The residual calculating circuit 21 calculates the residuals S(l, m) of the total number of L/(2a).times.M/(2b) from the picture element data Ak supplied from the representative point memory 11 through a switch SW2 to an input terminal 210 and the picture element data Bk(l, m) at the search points Qk(l, m) in the search areas Wk of the picture element data Bij of the second field supplied to the input terminal 211, and stores an intermediate result or a final result of the residuals S(l, m) in a residual memory 22.
The minimum one S(l0, m0) of the L/(2a).times.M/(2b) residuals S(l, m) as the final result stored in the residual memory 22 is detected by a movement vector calculating circuit 23, which calculates the movement vector Vp from a distance and a direction from the representative point Pk as the search origin to the position Qk(l0, m0) of the minimum residual S(l0, m0). Then, the movement vector Vp thus obtained is output to a control circuit 51.
In reading the picture signals of the second field stored in the field memory 41, the control circuit 51 adds a cumulated value of all the past movement rectors to the movement vector Vp supplied from the movement vector calculating circuit 23. According to the result of the addition, the control circuit 51 applies to the field memory 41 a reading address RA for correcting the picture movement due to the camera shake of the second field with respect to the first field. In this way, the picture signals of the second field obtained by correcting the picture movement due to the camera shake with respect to the picture of the first field according to the movement vector Vp are output from the field memory 41.
In the conventional construction mentioned above, the switch SW1 is operated in such a manner that the picture element data at the representative points in the odd fields are stored into the representative point memory 11 and the picture element data at the representative points in the even fields are stored into the representative memory 12. The switch SW2 is operated reversely to the above of the switch SW1, and supplies the picture element data at the representative points in the just previous field. Such a two-bank system that the representative point memory is constituted of the two representative point memories 11 and 12 is an ordinary means for effecting real-time image processing.
In the conventional construction mentioned above, the residual calculating circuit 21, the residual memory 22 and the movement vector calculating circuit 23 constitute a movement vector detecting circuit 20.
In most cases, the representative memories 11 and 12, the movement vector detecting circuit 20, the control circuit 51 and the switches SW1 and SW2 are constructed as a one-chip camera shake correcting circuit.
In the conventional camera shake correcting device as mentioned above, it is demanded to improve a correction accuracy. That is, it is demanded to improve an accuracy of the movement vector. However, in case of improving the accuracy of the movement vector three times, for example, the total number of the residuals S(l, m) to be calculated is increased nine times (=three times (H).times.three times (V)) as apparent from the above description. Accordingly, the residual memory 22 is required to have a memory capacity increased nine times. As a result, it is difficult to construct the camera shake correcting circuit as a one-chip circuit, causing an obstacle to provision of a compact and light video camera having a high function at a low cost.